Systems and methods for processing images of objects using interpolation between keyframes

ABSTRACT

An image processing system and method for determining an intrinsic color component of one or more objects present in a sequence of frames, for use in rendering the object(s), are described. Some of the frames of the sequence are to be used as lighting keyframes. A lighting estimate for a lighting keyframe A of the sequence of frames is determined. A lighting estimate for a lighting keyframe B of the sequence of frames is determined. A lighting estimate for an intermediate frame positioned between the lighting keyframes A and B in the sequence is determined by interpolating between the lighting estimates determined for the lighting keyframes A and B of the sequence. The determined lighting estimate for the intermediate frame is used to separate image values representing the object(s) in the intermediate frame into an intrinsic color component and a shading component, for use in rendering the object(s).

BACKGROUND

An image of one or more objects in a scene can be captured from theviewpoint of a camera. For example, the image may be a visual image,e.g. representing the visual appearance of the objects in the scene,e.g. in a format using Red, Green and Blue (RGB) values for pixels ofthe image, or in a format using luma and chrominance values (e.g. YUV).In some cases there may be more than one camera capturing differentimages of a scene. Each image of the scene represents a view of thescene from the viewpoint of the respective camera. The images mayrepresent frames of a video sequence.

As well as capturing the visual input images, depth images may becaptured representing the distances to points in the scene from thecamera as a function of pixel position. Depth cameras for capturingdepth images are known in the art, and may for example work byprojecting a pattern of infrared light into a scene and inferring depthfrom the disparity introduced by the separation between projector andsensor (this is known as a structured light approach). Alternatively,depth cameras may use a time of flight approach to determine depths bymeasuring the time taken for rays of infrared light to reflect back tothe sensor using interference, and from this inferring the depth ofpoints. As another alternative, depth images can be acquired from ascene reconstruction which is registered to the scene, given knowledgeof the camera calibration, for example by rendering the distance topoints in the scene by means of a depth buffer.

Images are produced by the interaction of light with the surfaces ofobjects in a scene. If the surface properties that produce an image, orset of images, can be found, then the image of the scene can bemanipulated (e.g. relit under arbitrary lighting conditions) usingconventional computer rendering techniques. Albedo (which may bereferred to as “intrinsic colour”), shading, surface normals andspecularity are examples of intrinsic surface properties, and techniquesthat estimate these from one or more images are known in the art as“intrinsic image methods”. Similarly, the extension to video is known as“intrinsic video”. It can help to simplify the problem of estimating theintrinsic surface properties to assume that the objects are non-specularand that the scene lighting is diffuse.

Input images captured by cameras have implicit real-world lightinginformation, such that lighting artefacts are present (i.e. “baked-in”)in the images. In order to relight the objects shown in an input image,an image processing system can attempt to split the image values (i.e.pixel values) of an input image into a shading component and anintrinsic colour component of the objects in the image. The intrinsiccolour component can be used for rendering the objects under differentlighting conditions. The splitting of the image values into shadingcomponents and intrinsic colour components is not a simple task.Therefore, typically, such image processing is performed “off-line” in apost-processing step after the images have been captured because theamount of time and/or processing resources required is typically large.Furthermore, this image processing is normally limited to static scenes,rather than performed on video sequences of moving objects.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

There is provided an image processing system configured to determine anintrinsic colour component of one or more objects present in a sequenceof frames, for use in rendering the one or more objects, wherein some ofthe frames of the sequence are to be used as lighting keyframes, theimage processing system comprising: lighting determination logicconfigured to: determine a lighting estimate for a lighting keyframe Aof the sequence of frames; determine a lighting estimate for a lightingkeyframe B of the sequence of frames; and determine a lighting estimatefor an intermediate frame positioned between the lighting keyframes Aand B in the sequence, by interpolating between the lighting estimatesdetermined for the lighting keyframes A and B of the sequence; andintrinsic colour determination logic configured to use the determinedlighting estimate for the intermediate frame to separate image valuesrepresenting the one or more objects in the intermediate frame into anintrinsic colour component and a shading component, wherein theintrinsic colour component is for use in rendering the one or moreobjects.

There is provided a method of determining an intrinsic colour componentof one or more objects present in a sequence of frames, for use inrendering the one or more objects, wherein some of the frames of thesequence are to be used as lighting keyframes, the method comprising:determining a lighting estimate for a lighting keyframe A of thesequence of frames; determining a lighting estimate for a lightingkeyframe B of the sequence of frames; determining a lighting estimatefor an intermediate frame positioned between the lighting keyframes Aand B in the sequence, by interpolating between the lighting estimatesdetermined for the lighting keyframes A and B of the sequence; and usingthe determined lighting estimate for the intermediate frame to separateimage values representing the one or more objects in the intermediateframe into an intrinsic colour component and a shading component,wherein the intrinsic colour component is for use in rendering the oneor more objects.

There may be provided computer readable code adapted to perform thesteps of any of the methods described herein when the code is run on acomputer. The computer readable code may be encoded on a computerreadable storage medium.

Furthermore, there may be provided a method of manufacturing, at anintegrated circuit manufacturing system, an image processing systemaccording to any of the examples described herein. In particular, theremay be provided an integrated circuit definition dataset that, whenprocessed in an integrated circuit manufacturing system, configures thesystem to manufacture an image processing system according to any of theexamples described herein. For example, the image processing system maybe embodied in hardware and configured for performing any of the methodsdescribed herein. The integrated circuit definition dataset may bestored on a computer readable storage medium.

The above features may be combined as appropriate, as would be apparentto a skilled person, and may be combined with any of the aspects of theexamples described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Examples will now be described in detail with reference to theaccompanying drawings in which:

FIG. 1 shows a device capturing images of objects;

FIG. 2 shows a first image processing system;

FIG. 3 is a flow chart for a method of determining an intrinsic colourcomponent and surface normals for one or more objects using the firstimage processing system;

FIG. 4 shows an arrangement of cameras capturing images of objects;

FIG. 5 shows a second image processing system;

FIG. 6 is a flow chart for a method of determining an intrinsic colourcomponent and surface normals for one or more objects using the secondimage processing system;

FIG. 7 illustrates processing of lighting estimates for a sequence offrames;

FIG. 8 is a flow chart for a method of determining lighting estimatesfor frames of a video sequence;

FIG. 9 shows a computer system in which an image processing system isimplemented; and

FIG. 10 shows an integrated circuit manufacturing system for generatingan integrated circuit embodying an image processing system.

The accompanying drawings illustrate various examples. The skilledperson will appreciate that the illustrated element boundaries (e.g.,boxes, groups of boxes, or other shapes) in the drawings represent oneexample of the boundaries. It may be that in some examples, one elementmay be designed as multiple elements or that multiple elements may bedesigned as one element. Common reference numerals are used throughoutthe figures, where appropriate, to indicate similar features.

DETAILED DESCRIPTION

Embodiments will now be described by way of example only. Embodimentsdescribed herein estimate the diffuse albedo and surface normals of thesurfaces of objects in a scene. The estimated albedo and surface normalsof the objects can be used for many different purposes, for example tofacilitate applications such as relighting, scene compositing,stylisation and illumination-independent appearance editing.

Examples described herein involve determining intrinsic colourcomponents and surface normals of objects in input images in a mannerwhich can be implemented in “interactive-time” (or “online”), e.g. asframes of a video sequence are captured. This can be achieved withoutusing vast processing resources due to optimisations used in the methodsdescribed herein. This means that examples described herein can beimplemented on mobile devices, such as smart phones, tablets, cameras,and laptops, as well as on other devices such as PCs. The performance ofthe method depends on the implementation, so “real-time” performancecould be achieved in some implementations. The appearance of the sceneunder novel lighting can be reproduced using the albedo and surfacenormals of objects in the input images together with arbitrary lightingconditions. The capture, intrinsic decomposition and relighting stagescan be performed simultaneously as a video processing pipeline operatingon multiple frames of a video sequence. By rendering and displaying theimages in “interactive-time” a user perceives the displayed images asshowing the current state of the objects but with different lightingconditions. For example, some of the examples described herein includeestimating full-resolution albedo (i.e. intrinsic colour components),shading, surface normals and lighting of dynamic scenes from RGBD dataat near real-time frame rates. RGBD data is a combination of visual data(i.e. RGB data) from one or more input images and depth data (i.e. D)from a depth image.

FIG. 1 shows objects 102 in a scene and a device 104 which is arrangedto capture images of the objects 102. The device 104 comprises a camera106 for capturing input images. The input images comprise image values(e.g. intensity values) representing a visual appearance of the objects102. In the examples described herein the image values are in an RGBformat, but it is to be understood that generally the image values maybe in any suitable format, such as YUV. The device also comprises adepth camera 108 for capturing depth images, a processing block 110arranged to perform image processing, and a store 112 arranged to storedata.

FIG. 2 shows components of the processing block 110 in more detail. Thecomponents of the processing block 110 may be implemented in hardware,software, firmware, or any combination thereof. The processing block 110comprises depth processing logic 202, surface normal logic 204, lightingdetermination logic 206 and intrinsic colour determination logic 208.The depth processing logic 202 comprises a filter 212; the surfacenormal logic comprises a filter 214; and the intrinsic colourdetermination logic 208 comprises a filter 216. The processing block 110may also comprise scene analysis logic 210. The scene analysis logic 210is shown with a dashed line because in some examples scene analysislogic is not implemented in the processing block. The processing block110 also comprises render logic 218 configured to render an imageincluding a view of the objects 102 under render lighting conditions.

A method of operating the image processing system at the device 104 isdescribed with reference to the flow chart shown in FIG. 3. In step S302the camera 106 captures an input image representing a view of theobjects 102. The input image is passed to, and received by, theprocessing block 110. Also, in step S302 the depth camera 108 captures adepth image representing depth values of the objects 102. The depthimage is passed to, and received by, the processing block 110. Cameras,such as camera 106, for capturing visual images (i.e. visual datarepresenting visible appearance) are well known in the art, and as suchthe details of how the camera 106 operates are not described herein.Depth cameras, such as depth camera 108, are also well known in the art,and may for example make use of time-of-flight and structured-lightdepth sensors, and as such the details of how the depth camera 108operates are not described herein. The “depth” values in the depth imagerepresent distances from the depth camera 108 to surfaces in the scene.The positions of the values of the input image and the depth imageapproximately match each other, but due to the sensors of the camera 106being located at a different position to the sensors of the depth camera108, a parallax effect may cause slight differences in the positions ofthe values from the input image and the depth image. Furthermore, thedepth image (D) may be of lower quality than the input image (T). Inparticular, depending on the depth camera 108, the depth image (D) maybe noisy and have quantisation effects, e.g. due to limited depthresolution, and the depth image may have a mismatched spatial resolutioncompared with the input image (T), and have object boundarymisalignments compared with the input image due to differences inviewpoint or calibration errors. Therefore, as described in more detailbelow in steps S304 to S308 the depth values and the surface normals(determined from the depth values as described below) are refined (i.e.upsampled and smoothed) to obtain refined depth values and refinedsurface normals which match the colour image as closely as possible. Inother examples, the depth values of the depth image may be determined inother ways (e.g. not using a depth camera). For example, if positions ofgeometry in the scene are known then the depth values can be determinedfrom the geometry, e.g. by rendering the distance to points of thegeometry in the scene using a depth buffer. The input image and therefined depth image are sufficiently aligned with each other to beconsidered to provide RGBD data, i.e. red, green, blue and depth valuesfor a number of pixel positions of the input image.

The pixel values of the input image, T(x) can be represented as theproduct of an intrinsic colour value ρ(x) at the pixel position x andthe irradiance I(x,n), i.e.:T(x)=ρ(x)I(x,n).  (1)

In a Lambertian scene, shading is equivalent to the irradiance, which isthe total incident light energy as a function of pixel position x andsurface normal n as shown in equation 2 below. The terms “shading”,“irradiance” or “lighting” may be used herein depending on the context,and may be represented as I. The term “lighting” refers to the placementof lights in a scene, and the term “shading” refers to the effect ofthose lights on the surface of an object.I(x,n)=∫_(Ω) R(x,ω)max(0,ω^(T) n)dΩ  (2)

The integral in equation 2 is performed over the sphere of unit vectorsΩ. R(x,ω) is the lighting as a function of surface position x anddirection ω. It is noted that x is used herein to represent surfacepositions, and their projections into the images (i.e. theircorresponding image positions). Furthermore, it is noted that x can beused herein to denote both a surface position and a pixel position,where the pixel position x is a projection of the surface position xinto screen space. This can be done because there is a one-to-onecorrespondence between visible points on the surfaces in the scene andtheir projections into screen space. In other words, given a surfacepoint on an opaque surface, there is no ambiguity about its projectioninto any cameras that have line-of-sight to it, and no ambiguity betweenpoints in images and their projections onto the scene surface. The termmax(0,ω^(T)n) is the clamped cosine kernel. To make the illuminationestimation problem tractable in the available time, the lighting isassumed to be position invariant (i.e. the same at all points in thescene) to a first approximation, and it is assumed that the observedinter-reflection and occlusion are negligible. Under these conditions,the irradiance is no longer dependent on surface position x, such thatequation 2 simplifies to:I _(G)(n)=∫_(Ω) R(ω)max(0,ω^(T) n)dΩ.  (3)

Since Lambertian scenes are assumed throughout, the global irradiancefunction I_(G) is estimated and used for illumination. I_(G) may bevisualised as a lighting function over Ω, which can be sampled by thesurface normal. The underlying radiance R(ω) does not have to beestimated, since Lam bertian reflectance is assumed and occlusion isneglected. Once I_(G) is estimated, it can be sampled by surface normalto arrive at a shading image (i.e. a set of shading estimates I(x) atpixel positions x.

The depth values of the depth image are received by the surface normallogic 204. In step S304 the surface normal logic 204 determines coarsesurface normal estimates (n_(C)) based on the depth values of thereceived depth image. In particular, the surface normal estimates(n_(C)) are determined based on differences between depth values of thedepth image. For example, the coarse surface normals n_(C) may berecovered from the depth values of the depth image using a finitedifferencing approach. For example, a finite difference method can beused to determine a 2D gradient of the surface of the objects 102 at thepositions of the depth values of the depth image. The coarse surfacenormals n_(C) are perpendicular to the 2D gradients determined from thedepth values of the depth image. In particular, the depth image isconverted to a 3D point cloud because the surface normals n_(C) aredetermined in “world space”, and then direction vectors between verticaland horizontal neighbours are found (i.e. finite differencing isapplied), and then the surface normals n_(C) are found as thecross-product of those vectors.

In steps S306 and S308, in order refine the coarse surface normalestimates and the depth values from the depth image, a type of bilateralfiltering is used to upsample and smooth the depth and surface normalestimates. Bilateral filtering such as this may be referred to as“cross-bilateral filtering” because it uses luma values derived from theinput image T as bilateral guidance terms. In particular, in step S306,the surface normal logic 204 uses the filter 214 to determine a set ofrefined surface normals by applying bilateral filtering to the coarsesurface normal estimates (n_(C)) using bilateral filtering guidanceterms based on luma values derived from the input image (T). Forexample, the surface normal logic 204 may determine refined surfacenormals n(x) for pixel positions x of the input image T(x), according tothe equation 4:

$\begin{matrix}{{n(x)} = {\frac{1}{u(x)}{\int{{n_{C}(y)}e^{- \frac{{{x - y}}_{2}^{2}}{\sigma_{d}^{2}}}e^{- \frac{{({l{({{T{(x)}},{T{(y)}}})}})}^{2}}{\sigma_{l}^{2}}}{dy}}}}} & (4)\end{matrix}$

In equation 4, u is a normalising term so that the filter weights arenormalised, and the first exponential term is a standard Gaussianspatial weighting term which varies according to the Euclidean distancebetween pixel positions x and y, where σ_(d) is variance. The secondexponential term in equation 4 represents the bilateral guidance basedon the luma values derived from the input image T, where l(T(x),T(y))represents the difference in luma values between pixel positions x and yof the input image, and σ_(l) is a variance term for a Gaussiandistribution of the luma values of the input image T. For example, wherethe input image T(x) is in an RGB format, then a luma value L(x) atpixel position x may be determined asL(x)=0.299T_(R)(x)+0.587T_(G)(x)+0.114T_(B)(x), where T_(R)(x)represents the Red channel values of the input image at pixel positionsx, T_(G)(x) represents the Green channel values of the input image atpixel positions x, and T_(B)(x) represents the Blue channel values ofthe input image at pixel positions x. In this case:l(T(x),T(y))=0.299(T _(R)(x)−T _(R)(y))+0.587(T _(G)(x)−T_(G)(y))+0.114(T _(B)(x)−T _(B)(y)).  (5)

As is known in the art, in other examples, a luma value L(x) at pixelposition x may be determined differently, e.g. asL(x)=0.2126T_(R)(x)+0.7152T_(G)(x)+0.0722T_(B)(x). In general, anysuitable determination of luma values from the values of the input imagemay be used.

It is noted that the terms “luma” and “luminance” may be used herein torefer to the same property, and as such the terms may be usedinterchangeably. Although at a detailed technical level it is understoodthat the terms “luma” and “luminance” do have subtly different meanings,at a higher level (i.e. in a broader sense) they can be understood torefer to the same property. Similarly, the terms “chroma” and“chrominance” may be used herein to refer to the same property, and assuch the terms may be used interchangeably. Although at a detailedtechnical level it is understood that the terms “chroma” and“chrominance” do have subtly different meanings, at a higher level (i.e.in a broader sense) they can be understood to refer to the sameproperty.

The integral of equation 4 may be performed over all of the pixelpositions x, or over just a subset of the pixel positions, x. Forexample, the integral may be performed over a region of ±10 pixelpositions in each dimension (e.g. horizontally and vertically) centredon pixel position x.

The bilateral filtering of equation 4 smoothes and upsamples the surfacenormals but does not blur the surface normals over edges in the inputimage. That is, there will tend to be large changes in luminance overedges in the input image, such that the value of l(T(x),T(y)) will tendto be high where x and y are on different sides of an edge, so thesurface normal n(x) at position x will not be blurred by the values ofthe coarse surface normal n_(C)(y) at position y. In this way, similarsurface normals are assigned to nearby pixels with similar appearance.Since the bilateral filter 214 uses guidance terms based on the lumavalues of the input image, the surface normals are refined to be morealigned with edges in the input image. The luma values are used for thebilateral guidance because changes in surface normals tend to be wellaligned to changes in the luminance of the input image. That is, thereis a strong correlation between surface normal direction and shading,and filtering the coarse surface normals based on the luma values of theinput image exploits this correlation. In particular, detailed changesin surface normals tend to be better aligned to the luminance of theinput image than to the chroma values of the input image or the depthvalues of the depth image. This is particularly the case under theassumption of diffuse scene lighting (i.e. non-specular lighting).

The refined surface normals n(x) can be provided to the store 112 forstorage therein. The refined surface normals can be used subsequently bythe render logic 218 for rendering the objects 102 under render lightingconditions, which may be different to the lighting conditions present inthe input image. The refined surface normals are also provided to theglobal lighting determination logic 206.

In step S308 the depth processing logic 202 uses the filter 212 toderive refined depth values by applying bilateral filtering to theinitial depth values of the depth image using bilateral filteringguidance terms based on luma values derived from the input image (T).The initial depth values can be denoted D_(C)(x) to indicate that theyare coarse values, in a similar manner to the coarse surface normalsn_(C)(x) referred to above. For example, the depth processing logic 202may determine refined depth values D(x) for pixel positions x of theinput image T(x), according to the equation 6:

$\begin{matrix}{{D(x)} = {\frac{1}{u(x)}{\int{{D_{c}(y)}e^{- \frac{{{x - y}}_{2}^{2}}{\sigma_{d}^{2}}}e^{- \frac{{({l{({{T{(x)}},{T{(y)}}})}})}^{2}}{\sigma_{l}^{2}}}{dy}}}}} & (6)\end{matrix}$where u, l(T(x),T(y)), σ_(d) and σ_(l) are defined as described above.Therefore, the first exponential term is a standard Gaussian filteringterm which varies according to the distance between pixel positions xand y, and the second exponential term represents the bilateral guidancebased on the luma values of derived from the input image T. Thebilateral filtering of equation 6 smoothes and upsamples the initialdepth values but does not blur the depth values over edges in the inputimage. Since the bilateral filter 214 uses guidance terms based on theluma values derived from the input image, the depth values are refinedto be more aligned with edges in the input image. This helps to correctfor any misalignment between the depth image D and the input image Twhich may occur, e.g. due to parallax resulting from the differentphysical positions of the camera 106 and the depth camera 108.Therefore, similar depth values are assigned to nearby pixels withsimilar appearance. In addition to smoothing out the unwantedquantisation artefacts from the initial depth values, the filteringperformed by the depth processing logic 202 recovers features which arepresent in the input image T but which are absent in the depth image D,e.g. due to noise. The refined depth values are provided to theintrinsic colour determination logic 208.

It would be possible to refine the initial depth values D_(C) todetermine the refined depth values D and then to determine the surfacenormals n from the refined depth values. However, in the example methodsdescribed herein the coarse surface normals n_(C) are determined fromthe initial depth values D_(C) and then the surface normals are refinedto determine the refined surface normals n. Refining the surface normalsand depth values separately in this manner, provides more accuratesurface normals because bilateral filters tend to have a certain amountof leakage which causes distortions in depth even across luma edges inthe input image. Although these distortions tend to be imperceptible inthe refined depth image, they would corrupt surface normals derived fromthe refined depth image.

In step S310 the scene analysis logic analyses the input image T toestimate scene geometry and to segment the input image into a pluralityof regions (or “segments”) representing respective materials. Each ofthe segments represents a respective region of similar appearance in theinput image. Methods of segmenting the surfaces of the objects 102 inthe input image are known in the art (e.g. see “Efficient Graph-BasedImage Segmentation” by Felzenszwalb and Huttenlocher, InternationalJournal of Computer Vision 59, 2, pp. 167-181). As described in moredetail below, in some embodiments, the input image is not split intosegments, but in the first example described herein, the input image issplit into segments.

In step S312 the global lighting determination logic 206 determines, inaccordance with the values of the input image T(x) and the determinedset of surface normals n(x), a global lighting estimate I_(G)(n(x))which provides consistent corresponding intrinsic colour estimates forthe segments of the objects 102 from the input image T(x). That is, inorder to estimate the global lighting estimate I_(G) a constraint thatregions with the same surface normals should have the same lighting isapplied. In some examples there may be an additional implicit constraintdue to the segmentation: nearby regions with similar appearance shouldhave similar albedos. Where the input image has been segmented, arepresentative albedo (or “intrinsic colour estimate”) is estimated foreach segment. The intrinsic colour estimates are chosen to minimise theshading differences between segments with similar surface normals. Mostsegments will have overlaps with multiple other segments in surfacenormal space, allowing a solution to global irradiance to be found. Thesegmentation may be discarded once the irradiance has been found.

In this way, in accordance with the values of the input image T(x) andthe determined set of surface normals n(x), representative intrinsiccolour estimates ρ_(i) are determined for each of the regions i, whereinthe representative intrinsic colour estimates ρ_(i) are determined basedon matches between corresponding shading estimates from differentregions for overlapping surface normal directions. In particular, therepresentative intrinsic colour estimates ρ_(i) are determined bychoosing representative intrinsic colour estimates that minimise thedifferences in shading between different regions for overlapping surfacenormal directions. The global lighting estimate I_(G)(n(x)) can then bedetermined using the values of the input image T(x) and the determinedrepresentative intrinsic colour estimates ρ_(i) for the regions i.

As an example, the surfaces of the objects 102 are segmented intosegments i, forming a set of segments θ. The pixel values of the inputimage T_(i)(x) for segment i are related to an approximation of theirradiance I′_(i)(ρ_(i),n(x)) for the segment i according to theequation:

$\begin{matrix}{{{I^{\prime}}_{i}( {\rho_{i},{n(x)}} )} = \frac{T_{i}(x)}{\rho_{i}}} & (7)\end{matrix}$where ρ_(i) is an, as yet unknown, representative albedo of segment i. Aset of representative albedos ρ (where ρ={ρ_(i)}) are chosen to minimisethe difference in irradiance between regions with overlapping surfacenormals, e.g. according to equation 8:E(ρ)=Σ_(i)Σ_(j>i)[∫_(Ω) [I′ _(i)(ρ_(i),ω)−I′ _(j)(ρ_(j),ω)]Q_(i,j)(ω)dΩ] ²  (8)where Q_(i,j) is a binary support function representing the overlap insurface normal direction between segments i and j, such thatQ_(i,j)={n_(i)}∩{n_(j)}. The optimum set of representative albedos ρ* ischosen to minimise E(ρ), such that ρ*=argmin_(ρ)E(ρ) subject tomax(ρ)=1.

One way of determining the optimum set of representative albedos ρ* isto determine the null vector of the matrix G, where:

$\begin{matrix}{G = \begin{matrix}{\begin{pmatrix}{\Sigma_{i \neq 1}{\mathcal{g}}_{1i}^{2}} & {{- {\mathcal{g}}_{12}}{\mathcal{g}}_{21}} & \ldots & {{- {\mathcal{g}}_{1n}}{\mathcal{g}}_{n1}} \\{{- {\mathcal{g}}_{21}}{\mathcal{g}}_{12}} & {\underset{\;}{\overset{\;}{\Sigma_{i \neq 2}}}\;{\mathcal{g}}_{2i}^{2}} & \ldots & {{- {\mathcal{g}}_{2n}}{\mathcal{g}}_{n2}} \\\vdots & \vdots & \ddots & \vdots \\{{- {\mathcal{g}}_{n1}}{\mathcal{g}}_{1n}} & {{- {\mathcal{g}}_{n2}}{\mathcal{g}}_{2n}} & \ldots & {\underset{\;}{\overset{\;}{\Sigma_{i \neq n}}}\;{\mathcal{g}}_{ni}^{2}}\end{pmatrix}} & \;\end{matrix}} & (9)\end{matrix}$whereg _(ij)=∫_(Ω)

(ω)Q _(i,j) dΩ  (10)where

(n(x))=T_(i)(x) is the projection of T_(i) into Ω by surface normal.

In step S314 the global lighting determination logic 206 determinesapproximate shading estimates I′(x) by dividing the values of the inputimage T(x) by the representative intrinsic colour estimate ρ_(i)* forthe segment i at surface position x, as shown in FIG. 11:

$\begin{matrix}{{I^{\prime}(x)} = {U_{{i\epsilon}\;\Theta}\frac{T_{i}(x)}{\rho_{i}^{*}}}} & (11)\end{matrix}$

These approximate shading estimates I′(x) indicate approximate shadingestimates at the surface positions x on the surfaces of the objects 102.These approximate shading estimates I′(x) can be used with knowledge ofthe surface normals n(x) at the surface positions x to determine aglobal lighting estimate I_(G)(ω) as a function of direction ω.

In particular, in step S316 the global lighting determination logic 206fits a linear combination of spherical harmonic components Y_(m) ^(l) tothe approximate shading estimate I′ to thereby determine the globallighting estimate I_(G), where m is the spherical harmonic degreeand/the spherical harmonic order. In the examples described herein thelinear combination of spherical harmonic components includes sphericalharmonic components up to the second order, because this gives a goodapproximation to the true global irradiance without being toocomputationally complex to be performed on low-cost processors (e.g.which may be implemented in mobile devices such as smart phones, tabletsand cameras) as discussed in a paper by Ramamoorthi, R., and Hanrahan,P. 2001, “On the relationship between radiance and irradiance:determining the illumination from images of a convex Lambertian object”,Journal of the Optical Society of America A 18, 10, 2448. Usingspherical harmonic components up to the 2^(nd) order is sufficient underan assumption that the lighting varies spatially with low frequency.However, in other examples more or fewer spherical harmonic componentsmay be included.

The global lighting estimate I_(G) can therefore be represented as:I _(G)=Σ_(m=0) ²Σ_(l=0) ^(m) k _(m) ^(l) Y _(m) ^(l)  (12)where {k_(m) ^(l)} is a set of complex coefficients to be determined,and {Y_(m) ^(l)} are complex spherical harmonic functions over thesphere. There are nine degrees of freedom in the lighting model. Aharmonic image method is used to determine the coefficients {k_(m)^(l)}. {Y_(m) ^(l)} are functions of direction, but it is useful to havethese complex functions represented in the surface co-ordinates x.Therefore harmonic images Q_(m) ^(l)(x) are defined as:Q _(m) ^(l)(x) =Y_(m) ^(l)(n(x))  (13)

We can then define

ϵ

^(p×H) to be a matrix of vectorised harmonic images, where p is thenumber of pixels in the input image, and H is the number of harmonicimages (H=6 in this example).

is defined as a vectorised version of the approximate shading estimatesI′(x). Then the global lighting determination logic 206 determines a setof coefficients k (where k={k_(m) ^(l)}) which gives the best fit to theapproximate shading estimates I′(x), e.g. in a least squares sense,according to the equations:argmin_(k) ∥

k−

∥ ₂ ²k*=(

^(T)

)⁻

^(T)

  (14)where k* represents the set of coefficients {k_(m)} which gives the bestfit. Substituting these coefficients back into equation 12 gives theglobal lighting estimate I_(G)(ω) as a function of direction ω. Theglobal lighting estimate I_(G)(ω) is provided to the intrinsic colourdetermination logic 208. The global lighting estimate I_(G)(ω) is alsoprovided to the store 112 for storage therein. As described in moredetail below, the global lighting determination logic 206 may retrievethe global lighting estimate for previous frames from the store 112 toguide the estimated global lighting estimate for a given frame of thevideo sequence.

In step S318 the intrinsic colour determination logic 208 determinesshading estimates I_(S)(x) using the determined global lighting estimateI_(G)(ω) and the determined surface normals n(x), such that:I _(S)(x)=I _(G)(n(x)),  (15)where x represents a surface position on the one or more objects. Inother words, the shading estimates I_(S)(x) are determined by samplingthe reconstructed lighting I_(G)(ω) by surface normal n(x).

In previous methods, shading estimates can become ill-conditioned, andsensitive to background noise in very dark regions of the input image.This is a general problem in intrinsic image estimation, and previousmethods tend to either avoid images with very dark regions or introduceadditional regularisation. However, in the methods described herein, theavailability of surface normals means that a shading estimate can stillbe found, even in dark regions. Furthermore, segments mainly comprisedof dark pixels can be detected and removed prior to albedo balancing.Segments may also be discarded due to incomplete surface normal data.The shading reconstruction is then performed on a reliable subset of thesegments Θ. The harmonic images only contain data for pixels in theinput image that contribute to the approximate shading estimates I′. Theshading image I_(S) is produced by sampling the global illuminationI_(G) by surface normal. This gives a shading estimate for all pixelsfor which surface normal data is available, including dark regions.

The intrinsic colour component of the objects 102 can be determinedusing the values of the input image and the determined shading estimatesI_(S)(x). In step S320 the intrinsic colour determination logic 208determines coarse intrinsic colour estimates ρ_(c)(x) such that

${\rho_{c}(x)} = {\frac{T(x)}{I_{S}(x)}.}$

As described above, the shading estimate I_(S) is determined based onthe surface normals of the objects 102. However, there may be highfrequency shading effects that cannot be modelled with the surfacenormals and the global lighting estimate. For example, high-frequencygeometry below the resolution of the depth image may not appear in thesurface normal image, so the remaining shading may be extracted by othermeans, as described below.

In step S322 the intrinsic colour determination logic 208 refines thecoarse intrinsic colour estimates to determine the intrinsic colourcomponent ρ(x) for the image. This is done by applying bilateralfiltering to the coarse intrinsic colour estimates using bilateralfiltering guidance terms based on: (i) luminance values derived from theinput image, (ii) chrominance values derived from the input image, and(iii) the refined depth values derived from the received depth image. Asdescribed above, the depth values are refined by the depth processinglogic 202 by applying bilateral filtering to the depth values of thedepth image using bilateral filtering guidance terms based on the lumavalues derived from the one or more input images. The refined depthvalues are provided to the intrinsic colour determination logic 208.

In particular, the intrinsic colour (or “albedo”) refinement is based onthe local neighbourhood B(x) surrounding a pixel position x. The localneighbourhood B(x) may for example include a block of pixel positionsextending in two dimensions by ±10 pixel positions centred on positionx, but in other examples, the neighbourhood may be a different shapeand/or size.

The intrinsic colour component ρ(x) can be determined by finding thevalues of ρ(x) which minimise the cost function:

$\begin{matrix}{{E( {\rho,I} )} = {{\Sigma_{x}( {{\rho(x)} - {\Sigma_{{y\epsilon B}{(x)}}{w( {x,y} )}{\rho(y)}}} )}^{2} + {\Sigma_{x}( {\frac{T(x)}{I(x)} - {\rho(x)}} )}^{2}}} & (16)\end{matrix}$where w(x,y) is a distance function given by:

$\begin{matrix}{{w( {x,y} )} = {e^{- \frac{{({\theta{({{T{(x)}},{T{(y)}}})}})}^{2}}{\sigma_{\theta}^{2}}}e^{- \frac{{({l{({{T{(x)}},{T{(y)}}})}})}^{2}}{\sigma_{l}^{2}}}}} & (17)\end{matrix}$where the first exponential term in equation 17 provides an indicationof the chroma difference between pixels T(x) and T(y) of the inputimage, and the second exponential term in equation 17 provides anindication of the luma difference between pixels T(x) and T(y) of theinput image. In particular, l(T(x),T(y)) is defined in the same way asdescribed above, e.g. as shown in equation 5 as an example. Theθ(T(x),T(y)) term represents the angle between vectors in RGB colourspace of the colour values of the input image at positions x and y, andmay for example be defined as:θ(T(x),T(y))=cos⁻¹({circumflex over (T)}(x)^(T) {circumflex over(T)}(y)),  (18)and σ_(θ) is the variance of the colour angle term. {circumflex over(T)}(x) is the chrominance at position x represented using a normalisedvector in RGB space. The colour angle θ is a measure of chrominancedifference.

The variances of the luma and colour-angle terms (σ_(l) and σ_(θ)) maybe adapted to the local neighbourhood. The first summation term ofequation 16 “flattens out” regions of similar albedo, whereas the secondsummation term avoids the trivial result ρ(x)=0 for all x. The sameresult is achieved by minimising the first term only, and updating I as

$I = {\frac{T}{\rho}.}$This can be achieved by iterative application of a bilateral filter (asshown in equation 19 for the t^(th) iteration), modified to incorporatethe colour similarity term of equation 18.

$\begin{matrix}{{\rho_{t + 1}(x)} = {\frac{1}{u(x)}{\int{{\rho_{t}(y)}e^{- \frac{{{x - y}}_{2}^{2}}{\sigma_{d}^{2}}}e^{- \frac{{({l{({{T{(x)}},{T{(y)}}})}})}^{2}}{\sigma_{l}^{2}}}e^{- \frac{{({\theta{({{T{(x)}},{T{(y)}}})}})}^{2}}{\sigma_{\theta}^{2}}}{dy}}}}} & (19)\end{matrix}$A further improvement to the filtering can be achieved using the refineddepth values derived from the depth image. Since discontinuities indepth correspond to occlusion boundaries, they are likely to coincidewith discontinuities in albedo. Therefore the availability of a depthchannel can be exploited by adding a depth guidance term as shown inequation 20:

$\begin{matrix}{{\rho_{t + 1}(x)} = {\frac{1}{u(x)}{\int{{\rho_{t}(y)}e^{- \frac{{{x - y}}_{2}^{2}}{\sigma_{d}^{2}}}e^{- \frac{{({l{({{T{(x)}},{T{(y)}}})}})}^{2}}{\sigma_{l}^{2}}}e^{- \frac{{({\theta{({{T{(x)}},{T{(y)}}})}})}^{2}}{\sigma_{\theta}^{2}}}e^{- \frac{{{{D{(x)}} - {D{(y)}}}}_{2}^{2}}{\sigma_{D}^{2}}}{dy}}}}} & (20)\end{matrix}$where D(x) and D(y) are the refined depth values provided by the depthprocessing logic 202 at positions x and y, and σ_(D) is the variance ofthe refined depth values.

The use of the depth values as bilateral guidance terms to refine theintrinsic colour estimates provides better intrinsic colour estimates inthe sense that intrinsic colour is not blurred over boundaries in thedepth image. This is beneficial because boundaries in the depth imagetend to indicate boundaries between different objects in the image, andblending the intrinsic colour estimates over these boundaries may resultin artefacts in the intrinsic colour estimates. Using the depth valuesas bilateral guidance terms reduces these artefacts.

The results of the bilateral filtering according to equation 20 are usedas the final intrinsic colour component ρ of the objects 102, which issent to the store 112 in step S324 for storage therein, to be used forrendering the objects 102 under render lighting conditions, which may bedifferent to the lighting conditions present in the input image. Thefinal irradiance estimates I_(f) can be found according to

${I_{f}(x)} = {\frac{T(x)}{\rho(x)}.}$The final irradiance estimates may, or may not, be stored in the store112 since they are not needed for rendering the scene under differentlighting conditions, but they may be useful for other purposes. Asdescribed above, the surface normals determined in step S306 are storedin the store 112, and this may be done at the same time as storing theintrinsic colour component in the store 112 in step S324, or it may bedone at another time, e.g. at the time of determining the surfacenormals.

The first iteration of the filter represented by equation 20 takes thecoarse intrinsic colour estimates as the input, i.e. ρ₀(x)=ρ_(C)(x). Thefilter can be applied for one or more iterations to determine the finalintrinsic colour component. Although a marginal improvement is possibleby performing multiple iterations, the intrinsic colour estimatesconverge rapidly, and a single iteration tends to be sufficient unless ahigh level of accuracy is desired. The number of iterations performed isimplementation specific, and may be adapted depending on the qualitydesired for a particular use of the system.

The filter in equation 20 is sensitive to noise in dark regions of theinput image. Although plausible intrinsic colour components ρ result inthese cases, the noise will be amplified in the final lighting estimatesI_(f). In the worst case, the lighting is undefined where ρ is estimatedto be zero. For this reason, dark regions are biased towards the coarseshading estimate I_(S) to determine a refined final lighting estimateI′_(f)(x) using a weighted sum such that:I′ _(f)(x)=αI _(S)(x)+(1−α)I _(f)(x)  (21)where α is a coefficient which sets the weight in the sum and takes avalue:

$\begin{matrix}{\alpha = e^{- \frac{{\min{\lbrack{T{(x)}}\rbrack}}^{2}}{\gamma}}} & (22)\end{matrix}$

The value of the cut-off parameter γ can be determined experimentally.

The render logic 218 is used to render a view of the objects underrender lighting conditions using the surface normals n(x) and theintrinsic colour components ρ(x), as is known in the art. A lightingfunction I_(r)(ω) defines the render lighting conditions as a functionof direction ω and is provided to the render logic 218. In particular,the surface normals n(x) are used in combination with the lightingfunction I_(r)(ω) to determine shading estimates I_(S)(x) for thesurface positions x of the objects 102 according toI_(S)(x)=I_(r)(n(x)). Then image values T_(r)(x) of the rendered imageon the surfaces of the objects can be determined according toT_(r)(x)=I_(S)(x)ρ(x). These image values T_(r)(x) are pixel valuesrepresenting the objects 102 in the rendered image under the renderlighting conditions. The rendered image can be provided to a display,e.g. for display in real-time. In the examples described above, thescene analysis logic 210 splits the surfaces of the objects 102 intoregions which are spatially contiguous segments of the objects. In otherexamples, the scene analysis logic 210 might not be implemented in theprocessing block 110. In the examples described below with reference toFIGS. 4 to 6, the regions are not necessarily spatially contiguous andare determined by identifying pixels of the input image with albedodifferences below a threshold.

FIG. 4 shows an example in which multiple views of the objects 102 fromrespective viewpoints are captured using multiple cameras (404 ₁ to 404₈). In the example shown in FIG. 4 there are eight cameras, but in otherexamples there may be a different number of cameras capturing views ofthe objects 102. The cameras 404 may be implemented in separate devicesbut may be arranged to provide the images that they capture to aprocessing system which can process the multiple images together. Theprocessing system may be implemented in the same device as one or moreof the cameras 404, or may be implemented in a separate device to all ofthe cameras 404. The processing system is shown in FIG. 5.

The processing system comprises a processing block 510 and a store 512.The processing block 510 comprises surface normal logic 504, globallighting determination logic 506 and intrinsic colour determinationlogic 508. The intrinsic colour determination logic 508 comprises afilter 516. The processing block 510 also comprises render logic 518. Amethod of using the system shown in FIG. 5 is described with referenceto the flow chart shown in FIG. 6.

In step S602 multiple input images are received at the processing block510 from the cameras 404. The multiple input images are provided to thesurface normal logic 504, to the global lighting determination logic 506and to the intrinsic colour determination logic 508. A depth camera isnot used in the example described with reference to FIGS. 4 to 6, but inother examples, one or more depth cameras could be used in combinationwith multiple cameras 404. Rather than determining surface normals usinga depth image, in the example shown in FIG. 5, a multi-viewreconstruction of the surfaces of the objects comprising the scene,hereinafter the “scene geometry” of the objects 102 (based on themultiple input images) is used to determine the surface normals.

In step S604 the surface normal logic 504 analyses the input images toestimate scene geometry and determine the surface normal estimates ofthe objects 102 based on the scene geometry. Methods of analysingmultiple input images of a scene from different viewpoints to determinescene geometry are known in the art (for example see the paper byFurukawa and Ponce entitled “Accurate, Dense, and Robust Multi-ViewStereopsis”, Pattern Analysis and Machine Intelligence, IEEETransactions on (Volume: 32, Issue: 8)), and as such are not describedin detail herein. The surface normals determined by the surface normallogic 504 are provided to the global lighting determination logic 506and are sent to the store 512 for storage therein for use in renderingthe objects under render lighting conditions.

The input cameras 404 produce a set of input images {T_(c)}_(cϵC) whereC represents the set of input images. The c subscript will be droppedherein where single images are being processed. Each input image T has acorresponding chroma image K, for which each pixel is normalised in RGBspace:

$\begin{matrix}{{K(x)} = {\frac{T(x)}{{{T(x)}}_{2}}.}} & (23)\end{matrix}$

In step S606 the global lighting determination logic 506 determines aglobal lighting estimate I_(G) which provides consistent intrinsiccolour estimates for regions of the input images. This is achieveddifferently compared to in the examples described above with referencesto FIGS. 1 to 3. Although the input images could be segmented intosegments, in this example, rather than segmenting the surfaces of theobjects 102 into a plurality of spatially contiguous segments, an albedosimilarity function Γ is used to identify pixels in a localneighbourhood which are likely to have similar albedo. These identifiedpixels can be considered as a region in a local neighbourhood defined byΓ. These regions are not necessarily spatially contiguous. This is basedon an observation that nearby pixels with similar chroma or luma valuesare highly likely to share similar albedo. It is noted that the regionidentified by Γ changes with surface position, and so is not asegmentation in the usual sense of the term. As an example, the albedosimilarity function may be defined as:

$\begin{matrix}{{\Gamma( {x,y} )} = \{ \begin{matrix}{1,} & {{{if}\mspace{14mu}{{{K(x)} - {K(y)}}}_{2}^{2}} < \kappa} \\{0,} & {otherwise}\end{matrix} } & (24)\end{matrix}$where K is a chromaticity threshold which may be fixed or variable, e.g.κ may be a function that adapts to the local neighbourhood. The globallighting determination logic 506 determines the regions by identifyingpixels with chromaticity differences below the chromaticity threshold κ.In other examples, the albedo similarity function Γ may be determined ina different manner. For example, the albedo similarity function Γincludes indications that may identify a probability that a first pixel(x) and a second pixel (y) are part of the same region, where the albedosimilarity indications may be determined based on at least one of: (i)differences in luma values for the first and second pixels, and (ii)differences in chroma values for the first and second pixels. Forexample, the albedo similarity function Γ may be determined according toequation 24a:Γ(x,y)=log [P(ρ;|L(x)−L(y)|)]+log [P(ρ;|K(x)−K(y)|₂)] yϵN(x),  (24a)where P(ρ;b) is the probability, given b, of two pixels at positions xand y having the same albedo, L is luma, K is chroma, and N(x) is thelocal neighbourhood of x.

As described above, the global lighting estimate I_(G)(n(x),k) as afunction of direction is defined in equation 12 as a sum of sphericalharmonic components {Y_(m) ^(l)} up to the second order, where {k_(m)^(l)} is a set of complex coefficients to be determined. Furthermore,the global lighting function can be mapped to initial shading estimatesI_(S)(x,k) for each input image for surface positions x on the objects102 using the surface normals, such that I_(S)(x,k)=I_(G)(n(x),k)=,where k represents the set of spherical harmonic coefficients {k_(m)^(l)}. The shading estimates I_(S)(x,k) relate to corresponding initialintrinsic colour estimates ρ_(S)(x,k) according toT(x)=I_(S)(x,k)ρ_(S)(x,k). A cost function Φ(k) can be used to find thebest fit for the set of coefficients k with the aim of reducingdifferences in intrinsic colour components for regions defined by thealbedo cost function F. It is noted that nearby pixels with similarchroma and luma are likely to share the same albedo, and albedo islikely to be piecewise flat within a small local region. The shading cantherefore be chosen to minimise the differences between nearby pixelswith similar chroma, luma or a combination of both. In this way the setof coefficients k is determined on the basis that pixels from the sameregion (i.e. representing the same material) are likely to have similarintrinsic colour components. The cost function Φ(k) is given by:Φ(k)=Σ_(C)Σ_(xϵX)Σ_(yϵN(x))Γ(x,y)∥ρS(x,k)−ρ_(S)(y,k)∥₂.  (25)

Each colour channel is optimised separately by minimising this costfunction. To avoid the trivial solution k=0, the “DC” element for eachcolour channel is set to 1. In accordance with equation 25, the set ofcoefficients k which minimise the cost function Φ(k) are determinedbased on differences between intrinsic colour estimates for the regions.Minimising the cost function Φ(k) can be considered to be choosing theshading to maximise the similarity of intrinsic colour estimates inlocal regions. The determined coefficients k define the global lightingestimate I_(G) according to equation 12. In equation 25, N(x) is a localneighbourhood of the surface position x, and for example may be an 11×11window centred on the position x.

The global lighting estimate I_(G) is provided to the intrinsic colourdetermination logic 508. In step S608 the intrinsic colour determinationlogic 508 uses the global lighting estimate I_(G) to determine shadingestimates I_(S)(x) at the surface positions x, as described above suchthat I_(S)(x)=I_(G)(n(x)). In step S608 the intrinsic colourdetermination logic 508 also determines corresponding coarse intrinsiccolour estimates (ρ_(S)(x)) according to

${\rho_{S}(x)} = {\frac{T(x)}{I_{S}(x)}.}$

The coarse intrinsic colour estimates ρ_(S)(x) determined in step S608may retain high-frequency shading detail that cannot be modelled usingthe coarse scene reconstruction, as well as inter-reflection and/or castshadows. So in step S610 the intrinsic colour determination logic 508uses the filter 516 to apply bilateral filtering to the coarse intrinsiccolour estimates using bilateral guidance terms based on chroma and lumavalues derived from the input image and using albedo similarityindications (Γ) which identify whether pixels are part of the sameregion.

A simple filter for smoothing out regions of similar chroma, thusenforcing the piecewise-constant albedo assumption could be used, suchthat:

$\begin{matrix}{{\rho_{f}(x)} = {\frac{1}{u}{\int_{X}{{\rho_{S}(y)}{\Gamma( {x,y} )}e^{- \frac{{{x - y}}_{2}^{2}}{\sigma_{d}^{2}}}{dy}}}}} & (26)\end{matrix}$where ρ_(f)(x) are the filtered intrinsic colour estimates, u is anormalisation factor, and X represents all of the surface positions x ofthe surfaces of the objects.

Furthermore, inter-reflection poses a particular challenge forchroma-based intrinsic image methods, since it shifts the chroma ofpixels so that regions with similar albedo no longer necessarily sharesimilar chroma. For example, if a surface is of uniform albedo r, afterthe n^(th) bounce an incoming white ray of light will have colour r^(n),which has the effect of saturating the chroma. In practice, this meansthat regions with high inter-reflectance are isolated from the rest ofthe scene by the chroma term in the simple filter represented byequation 26, making cast shadow removal difficult using the method inequation 26. To address this problem, the albedo similarity term Γ(x,y)can be relaxed in regions with high likelihood of inter-reflection andcast shadows. Ambient occlusion is strongly correlated with cast shadowsand inter-reflection, since all three are caused by concave scenegeometry. An estimate of ambient occlusion is therefore a usefulindicator of inter-reflection and cast shadows. Therefore, the filtermay include indications of likelihood of inter-reflection or castshadows at particular positions x, where those indications mayincorporate an estimate of ambient occlusion.

Therefore the intrinsic colour determination logic 508 determines theambient occlusion of the scene. The scene geometry is rendered underwhite directional lighting from a large number (e.g. 512) of randomdirections. These renders are summed to produce a normalised ambientocclusion estimate, A(x). When filtering across large discontinuities inambient occlusion A, the luma term is also disabled to allow forfiltering across the boundary. Both this and the adaptive chromaticitythreshold κ are shown in equation 27.

$\begin{matrix}{{\rho_{f}(x)} = {\frac{1}{u}{\int_{X}{{\rho_{S}(y)}{\Gamma_{AO}( {x,y,{{A(x)} - {A(y)}}} )}e^{- \frac{{{x - y}}_{2}^{2}}{\sigma_{d}^{2}}}{dy}}}}} & (27) \\{{{where}\mspace{14mu}{\Gamma_{AO}( {x,y,a} )}} = \{ \begin{matrix}{{( {1 + \gamma^{2}} ){\Gamma( {x,y} )}},} & {{{if}\mspace{20mu} a}\; \leq 0} \\{{\Gamma( {x,y} )},} & {otherwise}\end{matrix} } & (28)\end{matrix}$

The experimentally-determined parameter γ relaxes the albedo similarityΓ if the pixel x is more occluded than pixel y. The different cases forpositive and negative a ensure that this is a one-way relaxation, whichresults in occluded pixels assuming the appearance of their unoccludedneighbours, whereas unoccluded pixels remain unaffected by occludedneighbours.

In the case of multiple input views, cross-image constraints can beintroduced to ensure a consistent intrinsic colour estimate across allimages. In this way, the intrinsic colour determination logic 208 can beconfigured to determine said intrinsic colour component to be consistentacross the multiple views. For example, this can be achieved byiterative application of the filter in equation 29, initialised withρ₀=ρ_(S).

$\begin{matrix}{{\rho_{n + 1}(x)} = {\frac{1}{u}{\Sigma_{{c\epsilon}\; C}\lbrack {{\beta( {c,x} )}{\int_{X}{{\rho_{n}^{c}(y)}\;{\Gamma_{AO}( {x,y,{{A(x)} - {A(y)}}} )}e^{- \frac{{{x - y}}_{2}^{2}}{\sigma_{d}^{2}}}{dy}}}} \rbrack}}} & (29)\end{matrix}$β is a blending function, which weights the contribution of each albedoimage ρ_(n) ^(c) from a set of C input views depending on directness ofview and line-of-sight to the point x.

The filtered intrinsic colour estimates ρ_(f)(x) are used as theintrinsic colour component for the objects 102. In step S612 theintrinsic colour component is sent from the intrinsic colourdetermination logic 508 to the store 512 for storage therein. Thesurface normals are sent from the surface normal logic 504 to the store512 for storage therein. Furthermore, the global lighting estimate I_(G)may be sent from the global lighting determination logic 506 to thestore 512 for storage therein. The intrinsic colour component and thesurface normals may be subsequently used, e.g. by the render logic 518,to render an image of the objects 102 under render lighting conditions,as is described above in relation to the render logic 218.

The methods described above relate to determining intrinsic colourcomponents and surface normals for a scene at a single time instance,even though in some cases multiple views of the scene are used. Themethods can be extended to determining intrinsic colour components andsurface normals for multiple frames of a video sequence. When performingintrinsic video estimation, a constraint that the albedo is consistentacross frames of the sequence can be enforced to ensure a temporallycoherent decomposition into albedo and shading. In some previous methodspoint correspondences are used to constrain albedo across a sequence offrames. In those previous methods, dynamic scenes would require pointtracking for example using an optical flow technique, which iscomputationally complex and slow to run on current hardware on mostlow-cost mobile devices such as smart phones, tablets and cameras.

Temporal consistency in albedo and shading is achieved by introducing atemporal prior on lighting: consecutive frames are expected to havesimilar lighting. In methods described herein, scene lighting I_(G) isestimated for frames of a video sequence, as opposed to directlyestimating shading for the frames. It is simple to constrain the scenelighting to be consistent across multiple frames of a video sequence.Then, when a consistent lighting estimate I_(G) has been determined fora frame, the image data for the frame can be decomposed into intrinsiccolour and shading as described above. For sufficiently high frame rates(e.g. above 2 Hz), lighting estimates do not need to be determined onevery frame, but rather on lighting keyframes of the sequence of frames,which can provide an important performance optimisation. There is atradeoff between number of lighting keyframes (generally the fewer, thebetter), lighting stability and adaptability.

FIG. 7 shows an example representing seven frames of a video sequence(702 ₀ to 702 ₆), wherein one in three of the frames is designated as alighting keyframe. For example, the frames 702 ₀, 702 ₃ and 702 ₆ arelighting keyframes in this example, which may be referred to as lightingkeyframes A, B and C. For each frame of the video sequence, one or moreinput images are received and a depth image may be received, asdescribed above, in order to determine lighting estimates I_(G) for theframes of the video sequence. That is, for each frame of the sequence,one or more input images are received representing one or more views ofthe objects 102 at a time instance corresponding to that frame.

With reference to the flow chart shown in FIG. 8, a method ofdetermining lighting estimates is now described. In step S802 the globallighting determination logic (206 or 506) determines an initial lightingestimate 704 ₀ for the lighting keyframe 0 (702 ₀). The initial lightingestimate I_(G) for lighting keyframe 0 is determined according to any ofthe examples described above. As described above, the initial lightingestimate I_(G) is defined in terms of spherical harmonic coefficients{k_(m) ^(l)} according to equation 12, and is stored in the store 112.The initial lighting estimate 704 ₀ for the first lighting keyframe 702₀ is used as the lighting estimate 706 ₀ for that lighting keyframe. Theglobal lighting determination logic (206 or 506) can go on to determinelighting estimates for subsequent lighting keyframes.

In particular, in step S804 the global lighting determination logic (206or 506) sets a time index v to a value of 1. In step S806 the globallighting determination logic (206 or 506) determines an initial lightingestimate 704 ₃ for the lighting keyframe 1 (702 ₃). The initial lightingestimate I_(G) for lighting keyframe 1 is determined according to any ofthe examples described above. As described above, the initial lightingestimate I_(G) is defined in terms of spherical harmonic coefficients{k_(m) ^(l)} according to equation 12.

Then in step S808 the global lighting determination logic (206 or 506)determines a refined lighting estimate 706 ₃ for the for lightingkeyframe 1 based on: (i) the initial lighting estimate (704 ₃) for thelighting keyframe 1, and (ii) the lighting estimate (706 ₀) for thelighting keyframe 0. In particular, the lighting determination logic(206 or 506) may determine the refined lighting estimate 706 ₃ as aweighted sum of the initial lighting estimate 704 ₃ and the lightingestimate for the previous lighting keyframe (706 ₀). Since the lightingestimates each comprise a linear combination of spherical harmoniccomponents (in accordance with equation 12), the lighting determinationlogic (206 or 506) can refine different spherical harmonic coefficients(k_(m) ^(l)) for respective spherical harmonic components (Y_(m) ^(l))independently to each other. FIG. 7 shows initial lighting estimates ascircles with close hatching, and shows lighting estimates which arerefined by other lighting estimates as circles with more sparsehatching.

Therefore, a level of temporal smoothness can be enforced by applyingsignal processing methods to the spherical harmonic lighting coefficientvector {k_(m) ^(l)}. In particular, for lighting keyframe v therefinement step S808 may be implemented as:k _(m) ^(l) [v]=βk _(m) ^(l) [v−1]+(1−β)k′ _(m) ^(l) [v]  (30)k _(m) ^(l)[0]=k′ _(m) ^(l)[0]  (31)where {k_(m) ^(l)[v]} is the set of spherical harmonic coefficientsrepresenting the refined lighting estimate (706) for lighting keyframev, and {k′_(m) ^(l)[v]} is the set of spherical harmonic coefficientsrepresenting the initial lighting estimate (704) for lighting keyframev. Equation 31 indicates that for the very first lighting keyframe inthe video sequence, i.e. if there are no lighting keyframes in thesequence prior to the lighting keyframe 0, then the lighting estimate706 ₀ for the lighting keyframe 0 is the same as the initial lightingestimate (704 ₀) for the lighting keyframe 0. The parameter β is aparameter defining the weights in the weighted sum of equation 30.Higher values of β provide more stable lighting and shading estimates,but adaptation to changes in lighting in the frames of the sequence isslower. The lighting determination logic (206 or 506) may set theweights of the weighted sum (by setting β accordingly) based on thecontent of the sequence of frames. For example, if the content does notinvolve rapid changes in lighting then a higher value for β may be setcompared to if the content involves more rapid changes in lighting.

The frequency at which lighting is updated (i.e. the number of frames ofthe video sequence which are treated as lighting keyframes) is balancedwith the computational overhead incurred in performing the lightingestimation. For example, all of the frames in the sequence may be usedas lighting keyframes. In other examples (such as the example shown inFIG. 7), some but not all of the frames in the sequence are used aslighting keyframes. In the example shown in FIG. 7, one frame in everythree frames of the sequence is a lighting keyframe. In other examples,more or less of the frames may be lighting keyframes. To give someexamples, there may be a lighting keyframe for every 2, 5, 10, 20, 50 or100 frames of the sequence.

The lighting estimate 706 ₃ for the lighting keyframe 1 is stored in thestore 112 to be used for determining lighting estimates for subsequentlighting keyframes. With reference to FIG. 7, the lighting estimates forthe lighting keyframes (e.g. 706 ₀ and 706 ₃) are used as the finallighting estimates 708 for those frames (e.g. 708 ₀ and 708 ₃).

In examples, when not all of the frames of the video sequence arelighting keyframes, such as in FIG. 7, in step S810 the lightingdetermination logic (206 or 506) determines lighting estimates forframes of the sequence by interpolating between the lighting estimatesdetermined for lighting keyframes. In particular, lighting estimates forintermediate frames of the sequence, positioned between the lightingkeyframes (v−1) and v in the sequence, are determined by interpolatingbetween the lighting estimates for the lighting keyframes (v−1) and v.For example, the interpolation may be linear interpolation (or any othersuitable interpolation such as cubic interpolation) of the respectivespherical harmonic coefficients k_(m) ^(l) between the lightingkeyframes. Linear interpolation between keyframes is a straightforwardand computationally inexpensive solution that avoids abrupt transitionsin lighting estimates. In this way the lighting estimates 708 ₁ and 708₂ for frames 1 and 2 are determined based on the lighting estimates 706₀ and 706 ₃ for the lighting keyframes 0 and 1. The determined lightingestimates (708 ₀ to (708 ₃) for the frames 0 to 3 are provided to theintrinsic colour determination logic (208 or 508).

In step S812, for each of the frames for which lighting estimates arereceived, the intrinsic colour determination logic uses the lightingestimates to separate image values representing the objects 102 in theframe into an intrinsic colour component ρ and a shading componentI_(S), wherein the intrinsic colour component is for use in renderingthe objects 102 under render lighting conditions. The operation of theintrinsic colour determination logic is described above in relation tothe single frame examples, and the intrinsic colour determination logicmay operate as described above for each frame of the video sequence. Thedetermined intrinsic colour components of the frames are stored in thestore 112 for subsequent use in rendering the objects 102 under renderlighting conditions.

In step S814 the lighting determination logic (206 or 506) determineswhether there are more lighting keyframes in the sequence. In otherwords, if there are N lighting keyframes in the video sequence then stepS814 determines whether v is less than N−1.

If v is less than N−1 then there are more lighting keyframes in thesequence, so the method passes to step S816, in which v is incremented,e.g. from v=1 to v=2 in the first iteration. The method then passes backto step S806, and the steps S806 to S814 are repeated. In this way aninitial lighting estimate 704 ₆ is determined for lighting keyframe 2(in step S806), which is then refined in step S808 to determine thelighting estimate 706 ₆, and then the lighting estimates 708 ₄ and 708 ₅are determined in step S810 by interpolating between the lightingestimates 706 ₃ and 706 ₆ for the lighting keyframes 1 and 2. The finallighting estimate 708 ₆ is the same as the lighting estimate 706 ₆ forthe lighting keyframe 2 (702 ₆). In step S812 the intrinsic colourdetermination logic uses the lighting estimates 708 ₄, 708 ₅ and 708 ₆to determine the intrinsic colour components representing the objects102 in the frames 4, 5 and 6 respectively.

It is noted that step S808 may involve obtaining the lighting estimatefor the lighting keyframe v−1 by retrieving the lighting estimate forthe lighting keyframe v−1 from the store 112.

A time delay (e.g. at least as long as the time between lightingkeyframes in the video sequence) is introduced such that when a steadystream of frames of a video sequence is being processed then it shouldbe possible to interpolate between lighting keyframes, i.e. v is lessthan N−1. However, the video sequence could terminate at any point. Whenlighting estimates have been determined for all the lighting keyframesin the sequence then in step S814 it is determined that v is not lessthan N−1. If the video sequence does not end on a lighting keyframe thenthe frames of the video sequence after the last lighting keyframe may bedetermined in the same manner that lighting estimates for lightingkeyframes are determined. Alternatively, the lighting estimate for thelast lighting keyframe could be used for all of the frames after thelast lighting keyframe in the video sequence, i.e. the lighting estimatemay be fixed at the last known lighting estimate of a lighting keyframe.At this point the lighting estimates I_(G) for the frames of thesequence have been determined and intrinsic colour components ρ for eachof the frames have been determined and stored in the store 112. Themethod then passes from step S814 to S818, such that the method ends. Amethod for rendering the objects under render lighting conditions can beimplemented using the intrinsic colour components, e.g. by render logic(218 or 518) as described above.

The rendered views of the objects under the render lighting conditionsmay be used for a number of different applications. For example, a scenemay be rendered so as to appear with different lighting conditions tothose present when the images are captured, e.g. with lighting from adifferent direction, with different coloured lighting, or with lightingrepresenting different times of day (e.g. early morning, noon, sunset,moonlight, etc.). As another example, a view of the objects 102 may beincorporated into a different scene, and the lighting applied to theobjects can be set to match the lighting of the scene into which theobjects are being incorporated. To give some examples, the scene may bea real-world scene or a computer generated scene. As another example,the intrinsic appearance of the objects may be changed to be included inthe rendered image. For example, if the objects represent a woodensurface, the intrinsic colour components of the objects can be adjusted(separately to the lighting effects in the image) to appear like adifferent type of surface, e.g. a surface made from a different materialsuch as metal, plastic or fabric. The altered intrinsic colour componentcan then be relit according to the render lighting conditions forinclusion in the rendered image. This can be useful, e.g. for a user tosee what objects would look like if they had a different intrinsiccolour. Example uses of the rendered views of the objects for are shadowmanipulation, video stylisation and material editing.

Some further optimisations may be used in the methods described hereinto achieve near real-time intrinsic video frame rates on consumergraphics hardware. The method can be split into three distinct partswith no overlap in processing: (i) data acquisition and preprocessing(e.g. performed by the depth processing logic, the surface normal logicand the scene analysis logic), (ii) lighting estimation and intrinsicimage refinement (e.g. performed by global lighting determination logicand intrinsic colour determination logic), and (iii) intrinsicimage-based automatic image processing (e.g. relighting) and display ofresults (e.g. performed by the render logic) using the intrinsic colourcomponents and surface normals determined in stages (i) and (ii). Thesethree parts of the method can be run in separate threads and combinedinto a pipeline, to achieve higher throughput and processor utilisationon multi-core processors. As an example, the data acquisition stagecaptures the RGB and depth images, for example from a live RGBD videofeed, and calculates refined surface normals and refined depth values.The second stage produces lighting, shading and albedo estimates fromthe filtered surface normal and depth images, which are relit in thefinal stage.

In the intrinsic image refinement stage, calculating the angles betweenvectors in RGB space (for determining θ according to equation 18) takesconsiderable processing time. Computing an inverse cosine is anexpensive operation, so in an optimised implementation, the Euclideandistance between the normalised vectors may be used as an alternativechromaticity difference measure. This gives results that closely matchthose for the colour angle at a fraction of the processing time.

In the examples described above the render logic is implemented in thesame device as the logic used to determine in the intrinsic colourcomponent and surface normals. In other embodiments, the render logiccould be implemented in a different device. Furthermore, the renderlogic could be implemented after a significant delay (e.g. minutes ordays, etc.) after the image has been processed to determine theintrinsic colour component and surface normals, such that the renderingis not performed in real-time as the image(s) are captured.

FIG. 9 shows a computer system in which the image processing systemsdescribed herein may be implemented. The computer system comprises a CPU902, a GPU 904, a memory 906 and other devices 914, such as a display916, speakers 918 and the camera 106 and depth camera 108. A processingblock 910 (corresponding to processing blocks 110 or 510) is implementedon the GPU 904. In other examples, the processing block 910 may beimplemented on the CPU 902. The components of the computer system cancommunicate with each other via a communications bus 920. A store 912(corresponding to stores 112 or 512) is implemented as part of thememory 906. As described above, the camera 106 and the depth camera 108can be used to capture input images and depth images respectively.Rendered images may be displayed on the display 916. In other examples,one or more of the camera 106, depth camera 108 and display 916 may beimplemented on a different device to the processing block 910.

Generally, any of the functions, methods, techniques or componentsdescribed above (e.g. the components of the processing blocks 110 and510) can be implemented in modules using software, firmware, hardware(e.g., fixed logic circuitry), or any combination of theseimplementations. The terms “module,” “functionality,” “component”,“block”, “unit” and “logic” are used herein to generally representsoftware, firmware, hardware, or any combination thereof.

In the case of a software implementation of some of the components ofthe processing blocks 110 and 510, those components represent programcode that perform specified tasks when executed on a processor. In oneexample, units and logic of the processing blocks 110 and 510 may beperformed by a computer configured with software in machine readableform stored on a computer-readable medium. One such configuration of acomputer-readable medium is signal bearing medium and thus is configuredto transmit the instructions (e.g. as a carrier wave) to the computingdevice, such as via a network. The computer-readable medium may also beconfigured as a non-transitory computer-readable storage medium and thusis not a signal bearing medium. Examples of a computer-readable storagemedium include a random-access memory (RAM), read-only memory (ROM), anoptical disc, flash memory, hard disk memory, and other memory devicesthat may use magnetic, optical, and other techniques to storeinstructions or other data and that can be accessed by a machine.

The software may be in the form of a computer program comprisingcomputer program code for configuring a computer to perform theconstituent portions of described methods or in the form of a computerprogram comprising computer program code means adapted to perform allthe steps of any of the methods described herein when the program is runon a computer and where the computer program may be embodied on acomputer readable medium. The program code can be stored in one or morecomputer readable media. The features of the techniques described hereinare platform-independent, meaning that the techniques may be implementedon a variety of computing platforms having a variety of processors.

Those skilled in the art will also realize that all, or a portion of thefunctionality, techniques or methods described herein may be carried outby a dedicated circuit, an application-specific integrated circuit, aprogrammable logic array, a field-programmable gate array, or the like.For example, the module, functionality, component, unit or logic (e.g.the components of the processing blocks 110 and 510) may comprisehardware in the form of circuitry. Such circuitry may includetransistors and/or other hardware elements available in a manufacturingprocess. Such transistors and/or other elements may be used to formcircuitry or structures that implement and/or contain memory, such asregisters, flip flops, or latches, logical operators, such as Booleanoperations, mathematical operators, such as adders, multipliers, orshifters, and interconnects, by way of example. Such elements may beprovided as custom circuits or standard cell libraries, macros, or atother levels of abstraction. Such elements may be interconnected in aspecific arrangement. The module, functionality, component, unit orlogic (e.g. the components of the processing blocks 110 and 510) mayinclude circuitry that is fixed function and circuitry that can beprogrammed to perform a function or functions; such programming may beprovided from a firmware or software update or control mechanism. In anexample, hardware logic has circuitry that implements a fixed functionoperation, state machine or process.

It is also intended to encompass software which “describes” or definesthe configuration of hardware that implements a module, functionality,component, unit or logic (e.g. the components of the processing blocks110 and 510) described above, such as HDL (hardware descriptionlanguage) software, as is used for designing integrated circuits, or forconfiguring programmable chips, to carry out desired functions. That is,there may be provided a computer readable storage medium having encodedthereon computer readable program code in the form of an integratedcircuit definition dataset that when processed in an integrated circuitmanufacturing system configures the system to manufacture an imageprocessing system configured to perform any of the methods describedherein, or to manufacture an image processing system comprising anyapparatus described herein. The IC definition dataset may be in the formof computer code, e.g. written in a suitable HDL such asregister-transfer level (RTL) code. FIG. 10 shows an example of anintegrated circuit (IC) manufacturing system 1002 which comprises alayout processing system 1004 and an integrated circuit generationsystem 1006. The IC manufacturing system 1002 is configured to receivean IC definition dataset (e.g. defining an image processing system asdescribed in any of the examples herein), process the IC definitiondataset, and generate an IC according to the IC definition dataset (e.g.which embodies an image processing system as described in any of theexamples herein). The processing of the IC definition dataset configuresthe IC manufacturing system 1002 to manufacture an integrated circuitembodying an image processing system as described in any of the examplesherein. More specifically, the layout processing system 1004 isconfigured to receive and process the IC definition dataset to determinea circuit layout. Methods of determining a circuit layout from an ICdefinition dataset are known in the art, and for example may involvesynthesising RTL code to determine a gate level representation of acircuit to be generated, e.g. in terms of logical components (e.g. NAND,NOR, AND, OR, MUX and FLIP-FLOP components). A circuit layout can bedetermined from the gate level representation of the circuit bydetermining positional information for the logical components. This maybe done automatically or with user involvement in order to optimise thecircuit layout. When the layout processing system has determined thecircuit layout it may output a circuit layout definition to the ICgeneration system 1006. The IC generation system 1006 generates an ICaccording to the circuit layout definition, as is known in the art. Forexample, the IC generation system 1006 may implement a semiconductordevice fabrication process to generate the IC, which may involve amultiple-step sequence of photo lithographic and chemical processingsteps during which electronic circuits are gradually created on a wafermade of semiconducting material. The circuit layout definition may be inthe form of a mask which can be used in a lithographic process forgenerating an IC according to the circuit definition. Alternatively, thecircuit layout definition provided to the IC generation system 1006 maybe in the form of computer-readable code which the IC generation system1006 can use to form a suitable mask for use in generating an IC. Thedifferent processes performed by the IC manufacturing system 1002 may beimplemented all in one location, e.g. by one party. Alternatively, theIC manufacturing system 1002 may be a distributed system such that someof the processes may be performed at different locations, and may beperformed by different parties. For example, some of the stages of: (i)synthesising RTL code representing the IC definition dataset to form agate level representation of a circuit to be generated, (ii) generatinga circuit layout based on the gate level representation, (iii) forming amask in accordance with the circuit layout, and (iv) fabricating anintegrated circuit using the mask, may be performed in differentlocations and/or by different parties.

The term ‘processor’ and ‘computer’ are used herein to refer to anydevice, or portion thereof, with processing capability such that it canexecute instructions, or a dedicated circuit capable of carrying out allor a portion of the functionality or methods, or any combinationthereof.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims. It will be understood that thebenefits and advantages described above may relate to one example or mayrelate to several examples.

Any range or value given herein may be extended or altered withoutlosing the effect sought, as will be apparent to the skilled person. Thesteps of the methods described herein may be carried out in any suitableorder, or simultaneously where appropriate. Aspects of any of theexamples described above may be combined with aspects of any of theother examples described to form further examples without losing theeffect sought.

The invention claimed is:
 1. An image processing system configured todetermine an intrinsic colour component of one or more objects presentin a sequence of frames, for use in rendering the one or more objects,wherein some of the frames of the sequence are to be used as lightingkeyframes, the image processing system comprising: lightingdetermination logic configured to: determine, as a linear combination ofspherical harmonic coefficients, a lighting estimate for a lightingkeyframe A of said sequence of frames; determine, as a linearcombination of spherical harmonic coefficients, a lighting estimate fora lighting keyframe B of said sequence of frames; and determine alighting estimate for an intermediate frame positioned between thelighting keyframes A and B in said sequence of frames, by interpolatingthe respective spherical harmonic coefficients between said determinedlighting estimates for the lighting keyframes A and B of said sequenceof frames; and intrinsic colour determination logic configured to usethe determined lighting estimate for the intermediate frame to separateimage values representing said one or more objects in the intermediateframe into said intrinsic colour component and a shading component,wherein the intrinsic colour component is for use in rendering said oneor more objects.
 2. The image processing system of claim 1 wherein theinterpolation is linear interpolation.
 3. The image processing system ofclaim 1 wherein the lighting determination logic is configured todetermine the lighting estimate for the lighting keyframe B by:determining an initial lighting estimate for the lighting keyframe B;and refining the initial lighting estimate based on the lightingestimate for the lighting keyframe A.
 4. The image processing system ofclaim 3 wherein the lighting determination logic is configured todetermine the refined lighting estimate for the lighting keyframe B as aweighted sum of (i) the initial lighting estimate for the lightingkeyframe B, and (ii) the lighting estimate for the lighting keyframe A.5. The image processing system of claim 3 further comprising: surfacenormal logic configured to determine a set of surface normals for theone or more objects in the lighting keyframe B; and scene analysis logicconfigured to determine a plurality of segments of the one or moreobjects in the lighting keyframe B; wherein the lighting determinationlogic is configured to: determine representative intrinsic colourestimates for each of the segments, in accordance with image values ofthe lighting keyframe B and the determined set of surface normals,wherein the representative intrinsic colour estimates are determined soas to minimise differences in corresponding shading estimates betweendifferent segments for overlapping surface normal directions; anddetermine the initial lighting estimate for the lighting keyframe Busing the image values of the lighting keyframe B and the determinedrepresentative intrinsic colour estimates for the segments.
 6. The imageprocessing system of claim 1 wherein the intrinsic colour component isfor use in rendering the one or more objects in a video sequence underthe render lighting conditions.
 7. The image processing system of claim1 wherein the image processing system is embodied in hardware on anintegrated circuit.
 8. A method of determining an intrinsic colourcomponent of one or more objects present in a sequence of frames, foruse in rendering the one or more objects, wherein some of the frames ofthe sequence are to be used as lighting keyframes, the methodcomprising: determining, as a linear combination of spherical harmoniccoefficients, a lighting estimate for a lighting keyframe A of saidsequence of frames; determining, as a linear combination of sphericalharmonic coefficients, a lighting estimate for a lighting keyframe B ofsaid sequence of frames; determining a lighting estimate for anintermediate frame positioned between the lighting keyframes A and B insaid sequence of frames, by interpolating the respective sphericalharmonic coefficients between said determined lighting estimates for thelighting keyframes A and B of said sequence of frames; and using thedetermined lighting estimate for the intermediate frame to separateimage values representing said one or more objects in the intermediateframe into said intrinsic colour component and a shading component,wherein the intrinsic colour component is for use in rendering said oneor more objects.
 9. The method of claim 8 wherein the interpolation islinear interpolation.
 10. The method of claim 8 wherein said determininga lighting estimate for a lighting keyframe B comprises: determining aninitial lighting estimate for the lighting keyframe B; and refining theinitial lighting estimate based on the lighting estimate for thelighting keyframe A.
 11. The method of claim 10 wherein the refinedlighting estimate for the lighting keyframe B is determined as aweighted sum of (i) the initial lighting estimate for the lightingkeyframe B, and (ii) the lighting estimate for the lighting keyframe A,wherein the method further comprises setting the weights of the weightedsum based on the content of the sequence of frames.
 12. The method ofclaim 10 wherein said determining an initial lighting estimate for thelighting keyframe B of the sequence of frames comprises: determining aset of surface normals for the one or more objects in the lightingkeyframe B; determining a plurality of segments from the lightingkeyframe B, each of the segments representing a respective region in thelighting keyframe B; in accordance with image values of the lightingkeyframe B and the determined set of surface normals, determiningrepresentative intrinsic colour estimates for each of the segments,wherein the representative intrinsic colour estimates are determined soas to minimise differences in corresponding shading estimates betweendifferent segments for overlapping surface normal directions; anddetermining the initial lighting estimate for the lighting keyframe Busing the image values of the lighting keyframe B and the determinedrepresentative intrinsic colour estimates for the segments.
 13. Themethod of claim 12 wherein said determining the initial lightingestimate for the lighting keyframe B comprises determining approximateshading estimates I′ using the image values of the lighting keyframe Band the determined representative intrinsic colour estimates for thesegments, and fitting a linear combination of spherical harmoniccomponents Y_(m) ^(l) to the approximate shading estimates I′ to therebydetermine the initial lighting estimate I_(G) for the lighting keyframeB.
 14. The method of claim 13 wherein the linear combination ofspherical harmonic components includes spherical harmonic components upto, and not above, the second order.
 15. The method of claim 10 whereinthe lighting estimates each comprise a linear combination of sphericalharmonic components, and wherein different spherical harmoniccoefficients for respective spherical harmonic components are refinedindependently of each other.
 16. The method of claim 8 furthercomprising receiving, for each frame of the sequence, one or more inputimages each representing a view of the one or more objects at a timeinstance corresponding to that frame.
 17. The method of claim 8 whereinfor each of the frames, a plurality of images represent a plurality ofviews of the one or more objects from respective viewpoints, and whereinsaid lighting estimate for the lighting keyframe A is determined basedon the plurality of images for the lighting keyframe A and wherein saidlighting estimate for the lighting keyframe B is determined based on theplurality of images for the lighting keyframe B.
 18. The method of claim8 wherein the determined intrinsic colour component of the one or moreobjects are for use in rendering the one or more objects under renderlighting conditions.
 19. A non-transitory computer readable storagemedium having stored thereon processor executable instructions that whenexecuted cause at least one processor to determine an intrinsic colourcomponent of one or more objects present in a sequence of frames, foruse in rendering the one or more objects, wherein some of the frames ofthe sequence are to be used as lighting keyframes, the determining anintrinsic colour component comprising: determining, as a linearcombination of spherical harmonic coefficients, a lighting estimate fora lighting keyframe A of said sequence of frames; determining, as alinear combination of spherical harmonic coefficients, a lightingestimate for a lighting keyframe B of said sequence of frames;determining a lighting estimate for an intermediate frame positionedbetween the lighting keyframes A and B in said sequence of frames, byinterpolating the respective spherical harmonic coefficients betweensaid determined lighting estimates for the lighting keyframes A and B ofsaid sequence of frames; and using the determined lighting estimate forthe intermediate frame to separate image values representing said one ormore objects in the intermediate frame into said intrinsic colourcomponent and a shading component, wherein the intrinsic colourcomponent is for use in rendering said one or more objects.